Submodular Mini-Batch Training in Generative Moment Matching Networks

Generative moment matching network (GMMN), which is based on the maximum mean discrepancy (MMD) measure, is a generative model for unsupervised learning, where the mini-batch stochastic gradient descent is applied for the update of parameters. In this work, instead of obtaining a mini-batch randomly, each mini-batch in the iterations is selected in a submodular way such that the most informative subset of data is more likely to be chosen. In such a framework, the training objective is reformulated as optimizing a mixed continuous and submodular function with a cardinality constraint. A Majorization Minimization-like algorithm is used to iteratively solve the problem. Specifically, in each iteration of the training process, a mini-batch is first selected by solving a submodular maximization problem, and then the mini-batch stochastic gradient descent is conducted. Our experiments on the MNIST and Labeled Faces in the Wild (LFW) databases show the effectiveness of the submodular mini-batch training in the GMMN frameworks.